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Competence-based Quality Assurance of University Education 
Lessons Learnt from the OECD-AHELO Feasibility Study 
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The discourse that an educated workforce is essential for a prosperous and 
sustainable economy in a knowledge-based society’ has invited heightened policy 
interest in higher education. In effect, many industrialized countries are approaching, 
or have already reached universal access, with the majority of their age cohorts 
enrolling in higher education. This drastic expansion of higher education has invited 
new challenges to the system, particularly in relation to quality assurance. 

The focus of this paper is to explore models of quality assurance in the age 
of higher education expansion. In particular, this paper will focus on the Assess¬ 
ment of Higher Education Learning Outcomes (AHELO) Feasibility’ Study carried 
out by the Organization for Economic Cooperation and Development (OECD). 
The study serves as a compelling model for improving the quality of higher educa¬ 
tion in two respects. Firstly, the AHELO Feasibility Study is a model showing how 
learning outcomes assessment can serve as an action scheme to engage faculty in 
generating a common understanding of competence frameworks. Secondly, it is a 
model showing that competence frameworks can be shared among diverse univer¬ 
sities if they are sufficiently abstract, but also that abstract competence frame¬ 
works can be articulated into concrete learning outcomes that are attainable for 
students within a given timeframe, and assessable by faculty through the use of 
relevant assessment tools. 

The experience of participating in the AHELO Feasibility Study demonstrated 
that external learning outcomes assessments can help faculty in reaching substan¬ 
tive and tangible understanding of the competence framework, and in substanti¬ 
ating attainable and assessable learning outcomes from the abstract competences. 
Such shared understanding is indeed the foundation of competence-based quality 
assurance. 
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Introduction 

The discourse that an educated workforce is essential for a prosperous and sustainable 
economy in a knowledge-based society has invited heightened policy interest in higher educa¬ 
tion. In industrialized and developing countries alike, higher education has become increasingly 
important in national agendas, as a system not only to create and disseminate knowledge, but 
also to harness the competence that will enable people to work together and solve problems in 
an increasingly unpredictable and interconnected world. Industry has contributed to this trend 
through their employment and wage policies that advantage higher education graduates. 

In effect, many industrialized countries are approaching, or have already reached universal 
access, with the majority of their age cohorts enrolling in higher education. In developing coun¬ 
tries, dramatic increases in higher education enrollment have been made possible by democra- 
tisation, urbanization, and the resulting growth in upper secondary education completion rates. 
According to the UNESCO Institute for Statistics, the number of students enrolled in higher 
education worldwide is estimated to have expanded from 32.5 million in 1970 to 178 million in 
2010 (UIS, 2012). 

The drastic expansion of higher education has invited new challenges to the system, partic¬ 
ularly in relation to quality assurance. The focus of this paper is to explore models of quality 
assurance in the age of higher education expansion. In particular, this paper will focus on the 
Assessment of Higher Education Learning Outcomes (AHELO) Feasibility Study carried out 
by the Organization for Economic Cooperation and Development (OECD). The study serves 
as a compelling model for improving the quality of higher education in two respects. Firstly, 
the AHELO Feasibility Study is a model showing how learning outcomes assessment can serve 
as an action scheme to engage faculty in generating a common understanding of competence 
frameworks. Secondly it is a model showing that competence frameworks can be shared among 
diverse universities if they are sufficiently abstract, but also that abstract competence frame¬ 
works can be articulated into concrete learning outcomes that are attainable for students within a 
given timeframe and assessable by faculty through the use of relevant assessment tools. 

The first section will outline the context of contemporary higher education, highlighting 
why competence-based quality assurance is important but difficult to pursue. The second section 
will articulate on higher education policy initiatives in Japan, showing how the focus on quality 
assurance has shifted over the past decade from inputs to outcomes, but how policy has been 
unsuccessful in providing concrete and constructive models for universities to reference. The 
third section will introduce the Japanese experience of participating in the AHELO Feasibility 
Study, showing how the international learning outcomes assessment successfully engaged faculty 
and contributed to the development, shared understanding, and utilization of global competence 
frameworks. Finally, the paper will conclude with a section on future directions for quality 
assurance in a knowledge-based global society. For simplification, this paper will focus only on 
university education at the bachelor-level, the largest sector of higher education in Japan. 


I.The Context of University Education in the 21 st Century 

1.1. The Growing Demand for Quality Assurance 

The emergence of new types of institutions serving students from diverse academic back- 
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grounds and with diverse career destinations calls for the redefinition of the role of higher 
education in society. Under particular consideration is the role of universities that have tradition¬ 
ally enjoyed elite status as academically oriented institutions, but are quickly transforming with 
the emergence of new professionally oriented institutions. What does it mean to be a college 
graduate? What are university students expected to know and be able to do by the comple¬ 
tion of their degree-programs? What are the competences that universities must assure when 
they confer degrees? As the costs of higher education increase, universities are being put under 
increasing pressure to be accountable for the outcomes of their educational activities. 

The increase in student mobility across institutions within countries and among countries 
has led to the recognition that quality assurance schemes can no longer function autonomously 
within individual institutions. There is a growing recognition that, if student exchange programs 
and joint degree-programs are to function fruitfully, a certain level of convergence, based on 
a common understanding of what competences college credits and degrees represent needs 
to be achieved. Institutions that collaborate in the provision of educational programs seek for 
quality assurance systems that facilitate the comparability and transferability of credits. Simi¬ 
larly, employers that hire globally seek quality assurance systems that facilitate the transparency 
and comparability of college degrees (Adelman, 2009). 

1.2. The Difficulties of Competence-based Quality Assurance 

The introduction of competence-based quality assurance schemes has invited much contro¬ 
versy among the higher education community largely for two reasons. First, the imposition of 
external competence-based quality assurance schemes poses a direct threat to universities that 
have historically enjoyed institutional autonomy and academic freedom. Universities are having 
a difficult time accepting the notion of governments or third party evaluation agencies defining 
what university students should know and be able to do by the completion of degree-programs. 
This is particularly controversial because the concept of “competence” originates in the employ¬ 
ment sector, calling for the alignment of knowledge and skills from the primary, secondary, 
and tertiary education into the workforce. The focus on alignment is conceived to shift focus 
away from creative intellectual skills that have traditionally been valued in universities, such as 
exploring new ideas, constructing new paradigms, and reconstructing existing systems, toward 
practical skills that are directly applicable to industry. 

Secondly, the idea of universities sharing a competence framework challenges the diversity 
that has allowed universities to expand. As stated earlier, elite universities are quite different 
from the new universities that have emerged as a result of massification, in their academic and 
professional profiles, the students they enroll, the career pathways their graduates pursue, etc. 
In reality, the comparability and transferability of credits and degrees are pursued among “peer” 
universities but not necessarily among system-wide institutions. The flourishing of formal univer¬ 
sity partnership agreements represents strategic actions by universities to identify themselves 
with universities of similar orientation. It is not yet clear how institutional diversity, which has 
allowed wider access to the system, can be maintained under a common competence framework. 

How can universities cope with the new challenges of a knowledge-based global society, 
without compromising their autonomy and diversity? How can governments organize quality 
assurance schemes that will build on the strengths of universities, but at the same time respond 
to societal expectations? How abstract must a competence framework be if it were to be shared 
by autonomous and diverse institutions? How concrete must it be if it were to provide mean- 
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ingful reference points to assure quality? These are critical questions that require careful consid¬ 
eration when exploring models of quality assurance. 


2. Higher Education Policy Initiatives in Japan 

2.1. The Shift from Inputs to Outcomes 

In response to the challenges of a knowledge-based global society, the Japanese Ministry 
of Education, Culture, Sports, Science and Technology (MEXT) has been actively promoting 
higher education reform by commissioning its advisory board, the Central Education Council, to 
examine issues relevant to higher education in a knowledge-based global society. The Council’s 
recommendations are endorsed by MEXT through competitive grants distributed to universities 
that undertake innovative reform in alignment with its principles. 

It is noteworthy to highlight a significant policy shift that took place between 2005 and 
2008. According to the Central Education Council report issued in 2005, The Future of Higher 
Education in Our Country, a knowledge-based global society calls for university responsive¬ 
ness to diverse societal needs. Seven diverse functions of universities were identified, including 
world class centers for research and education, comprehensive liberal arts education, community 
centers for life-long learning, among others. Universities were advised to identify with one or 
multiple of these diverse functions, and to develop their degree-programs accordingly. Quality 
assurance within this context meant assuring that the university system as a whole fulfills diverse 
societal missions (Central Education Council, 2005). 

In line with this principle, the approval system for university establishment which regu¬ 
lated educational inputs such as the organization, staffing, curriculum, facilities, and gradua¬ 
tion requirements was simplified to promote diversity. To compensate for the deregulation, a 
third party certified evaluation system was implemented in 2004, focusing on the effectiveness 
of internal quality assurance systems, and encouraging institutions to pursue diverse missions 
(NIAD-UE, 2009). 

Deliberations by the Central Education Council however led to a different concept of quality 
assurance as articulated in its 2008 report, Towards the Construction of Bachelor Degree-pro- 
grams. The Council emphasized the role of universities in fostering a common set of 21 st -cen- 
tury competences, called Gagushi Ryoku. It is a set of loosely defined transversal skills that 
bachelor-level students are expected to acquire, regardless of the field of study, including 1) 
knowledge and understanding, 2) generic skills for various purposes, 3) attitude and propensity, 
and 4) integrative learning experience and creative thinking 1 . Universities were advised to refer¬ 
ence Gakushi Ryoku when defining the competences that will be pursued in their degree-pro- 
grams. Universities were also advised to specify their student admissions policies, curriculum 
policies, and diploma conferment policies in alignment with pursued competences (Central 
Education Council, 2008). 

In parallel with the Central Education Council’s work on Gakushi Ryoku, and in response 
to MEXT’s request, the Science Council of Japan has been working on defining disciplinary 
competence frameworks (Science Council of Japan, 2010). Formal replies have been made, or 
are underway by faculties of business administration, linguistics and literature, law, mechanical 
engineering, and mathematical science, and other disciplines are slowly following their lead. 

The emphasis on the alignment of admissions-curriculum-diploma policies symbolises the 
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Japanese dilemma in quality assurance. Traditionally, Japanese universities have relied on rigid 
college entrance examinations to assure the college-readiness of their students. However, the 
drastic decrease in the number of the 18-year-olds has disenabled some universities to rely on 
this approach. 

Between the mid-1970s and late-1980s, the percentage of 18-year-olds “applying” to 
degree granting two and four-year institutions was stable at around 50%, with the percentage 
being “accepted” around 35%. However, the numbers have fluctuated over the past two decades, 
reaching 61% (applied) and 57% (accepted) in 2011. Over 90% of high school seniors that 
applied for two and four-year colleges were accepted (MEXT, 2012). 

The competitive university entrance examination system had given highly-ranked univer¬ 
sities the leisure of accepting college-ready students and maintaining a loosely-coupled educa¬ 
tional approach, in which they were able to postpone the institutionalization of systemic quality 
control processes, such as learning outcomes assessment, grade point average, and PDCA cycle. 
These universities have indeed allowed almost all of their students to graduate on time with 
degrees. However, due to the decrease in the 18-year-old population and the resulting number 
of applicants, some universities have had to accept any student that applies, regardless of their 
college-readiness, just to maintain enrollment and stay in business. The recent expansion of 
university first year and developmental courses is evidence of the magnitude of this problem. 
Within this context, universities have been required by MEXT to become more accountable for 
their education, through more careful planning on who they accept as students, how they teach 
them, and what they require upon graduation. 

2.2 The Lack of an Action Scheme to Induce System-wide Faculty Collaboration 

The most recent report issued by the Central Education Council in 2012, Towards the 
Qualitative Change of University Education for a New Future, basically echoes its 2008 report, 
calling for competence-based degree-program development. The report goes on to propose 
several concrete recommendations for qualitative change, such as increasing the amount of time 
students spend on autonomous study and expanding student support systems, which are related 
to, but not central to competence-based program development (Central Education Council, 
2012 ). 

The Science Council has also requested universities to reference the disciplinary compe¬ 
tence frameworks the Council has been promoting when designing their degree-programs. 
However, honouring the autonomy and diversity of universities, the Council has refrained from 
articulating on the actual process of program development, leaving institutions to explore indi¬ 
vidually through trial and error (Science Council of Japan, 2010). 

Indeed, the current Japanese higher education policy emphasizes the need for compe¬ 
tence-based quality assurance, and provides various ideas and competitive grants for trans¬ 
forming university education. However, inducing such committed “collective action” among 
faculties across institutions requires a well-planned action scheme, leadership, and coordination. 
Policy has so far not been successful in mobilizing faculty to engage in competence framework 
sharing and competence-based degree-program development. 

Indeed, the competence frameworks proposed by the Central Education Council and 
the Science Council may potentially serve as important starting points for discussion among 
faculty in designing degree-programs based on shared competence frameworks. However, much 
systemic work needs to be done before institutions at scale are able to embrace the frameworks, 
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and utilize them for educational improvement. Policy has so far not been successful in providing 
constructive options to guide this process. 


3. The AHELO Feasibility Study 

3.1. An Effective Model for Organizing Collective Action 

The OECD-AHELO is an international assessment of higher education learning outcomes 
that aims to assess what university students know and can do upon completion of their bachelor 
degree-programs. A “Feasibility Study” focusing on whether or not it is practically and scientif¬ 
ically feasible to conduct such an assessment was implemented between 2009 and 2012, in the 
strands of generic skills, economics, and engineering (in the field of civil engineering), involving 
17 countries, 248 institutions, and 23,000 students. Japan took part in the engineering strand of 
the Feasibility Study, with the participation of 12 institutions and 504 students. 

As an international experiment involving the collaborative activities of designing the 
competence framework, developing the assessment instrument and scoring rubrics, and scoring 
student responses, the AHELO Feasibility Study serves as a unique and effective model for 
bringing together faculty from diverse institutions to discuss and reach consensus on compe¬ 
tence frameworks and learning outcomes. How such committed “collective action” was made 
possible deserves special attention. 

The AHELO Feasibility Study also serves as an exemplary model of how competence 
frameworks may be shared by autonomous and diverse institutions when defined in sufficiently 
abstract ways, but at the same time provide substantive and tangible information on student 
learning when articulated into concrete and measurable learning outcomes. 

Focusing on the engineering strand of the AHELO Feasibility Study, the following sections 
will first document how the study was organized allowing for faculty to be engaged as central 
actors in each phase of the study, and secondly how that faculty engagement proved to be crit¬ 
ical in generating common understanding of what university students should know and be able 
to do upon the completion of their degrees. 

3.2. Involving Faculty as Central Actors 

It is important to focus on the organization of the management team for the AHELO Feasi¬ 
bility Study, in order to highlight the central roles faculty played in each phase of the study. The 
international AHELO Consortium was appointed in mid-2010 by OECD after a call for tender 
(bidding) process. Headed by the Australian Council for Educational Research (ACER), and 
constituted by diverse partner organizations shown in Diagram 1, the Consortium undertook 
responsibility over the instrumentation and implementation of the assessment 2 . 

Within the Consortium, the National Institute of Educational Policy Research of Japan 
(NIER), in partnership with ACER and the University of Florence School of Engineering (UF), 
undertook the role of developing an engineering assessment instrument. The task involved coor¬ 
dinating deliberations by the international engineering expert group, which consisted of engi¬ 
neering faculty members and industry professionals who have acted as national and global 
leaders in engineering education and have professional recognition. From Japan, two engineering 
professors participated as members of this expert group. 

In addition to this international engineering expert group that oversaw the Consortium’s 
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Diagram 1 The AHELO feasibility study management team 


work, each engineering team organization had in-country or regional expert groups to consult with 
as necessary. As will be elaborated later, NIER collaborated with a research team commissioned 
by MEXT, while ACER consulted with the Australian Council of Engineering Deans, and UF 
consulted with the European and Global Engineering Education academic network (EUGENE). 

In Japan, the decision to participate in the AHELO Feasibility Study was made in 2009 by 
MEXT, based on the recommendation from the Central Education Council AHELO Working 
Group. MEXT commissioned NIER to function as the National Center, responsible for the trans¬ 
lation and adaptation of the instruments, test administration, and scoring. 

MEXT also commissioned Research on Approaches to Implementing the OECD-AHELO 
Feasibility Study, hosted by the Tokyo Institute of Technology, and consisting of engineering 
faculty members from twelve diverse universities and higher education research experts. The 
primary purpose of the study was to generate scholarly understanding of AHELO, and to build 
consensus on how Japan would participate in the international effort. The team conducted 
comparative research on university quality assurance systems and standards, as well as student 
mobility schemes around the world, through international field research, lectures by guest 
























56 


Fukahori, Satoko 


speakers, and hours of discussion. 

Within the research team was created a small task force of engineering faculty members, 
which functioned as the advisory group for the National Center. This expert group engaged in the 
verification of instruments and scoring rubrics, verification of the translation and adaptation of 
instruments and scoring rubrics, and the actual scoring of student responses. While their inputs 
were crucial in improving the quality of instruments and scoring rubrics, their active involve¬ 
ment in these activities generated among the wider group of engineering faculty members strong 
interest and a sense of confidence in the study. 

It is important to note that this research-driven collaboration and involvement by faculty, 
which proved invaluable in developing deep understanding of the AHELO endeavour and the 
need for competence-based quality assurance, also generated strong engagement by individuals 
and institutions that care sincerely for the education of their students. The implementation of the 
AHELO Feasibility Study in Japan would not have been as fruitful without the commitment of 
this faculty-led research team (Tokyo Institute of Technology, 2012). 

3.3. Engineering Competence Framework Development 

The engineering competence framework was developed by an engineering expert group 
commissioned by OECD in consultation with the Tuning Association. According to the Tuning 
approach, competences are defined by faculty in consultation with stakeholders (employers 
and graduates), and represent dynamic combinations of knowledge, understanding, skills, and 
abilities. They describe the discipline’s nature and core concepts, and identify the learning that 
comprises the discipline at given degree levels. Autonomous and diverse universities are respon¬ 
sible for determining from among the core concepts which competences they will prioritize, and 
for developing degree-programs consisting of courses and modules that will ultimately guide 
students to acquiring the expected competences. 

Faculty members are responsible for defining the learning outcomes to be pursued in the 
courses or modules they teach. Learning outcomes are statements of what a learner is expected 
to know, understand, and able to demonstrate after completion of learning. Learning outcomes 
must be attainable by the student within a given timeframe and assessable through demonstra¬ 
tion, as they specify the requirements for the award of credits (Gonzales & Wagenaar, 2008; 
IEBC, 2012). 

This distinction between competences and learning outcomes has important implications 
for developing degree-programs based on shared competence frameworks. The competence 
framework must be abstract, describing the overarching core concepts of the discipline, and 
applicable to a diverse array of institutions. The competences must be articulated into concrete 
learning outcomes by faculty, guiding the structure of their degree-program, as well as the 
learning content, method, and assessment approaches of the courses or modules that constitute 
the degree-program. 

The Tuning-AHELO engineering competence framework adapted to be used for assessment 
in the civil engineering field is shown in Table 1. The bachelor-level engineering competences 
were defined as the demonstrated capacity to solve problems by applying principles of “basic 
and engineering sciences”, “engineering processes” and “engineering generic skills” (OECD, 
2011). The framework strongly resembles existing frameworks of the International Engineering 
Alliance’s Graduate Attribute Profile shared by accredited institutions in Washington Accord 
signatories, and the European Network for Accreditation of Engineering Education’s EUR-ACE 
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Table 1 The Tuning-AHELO engineering competences—for civil engineering 


Basic and Engineering Sciences 

i) The ability to demonstrate knowledge and understanding of the scientific and mathematical principles underlying 
their branch of engineering. 

♦ Mathematics: real and complex analysis, linear algebra, differential equations, Fourier series, Laplace transfoms, 
numerical methods, vector calculus, probability and statistics. 

♦ Sciences: general physics (including fields, waves, mechanics, matter, forces), electrical technology, electronics, 
material science and strength of materials, fluid mechanics. 

♦ Chemistry, engineering geology, technical mechanics, statics and continuum mechanics, structural mechanics, 
hydraulics. 

ii) The ability to demonstrate a systematic understanding of the key aspects and concepts of their branch of engi¬ 
neering. For Civil Engineering: 

♦ Building materials, environmental sciences, building physics, surveying, fundamentals of planning, structural 
theory, engineering drawing, operations research, introduction to GIS, architectural drawings, electro-mechanics 
for civil engineering, introduction to environmental engineering. 

iii) The ability to demonstrate comprehensive knowledge of their branch of engineering including emerging issues. 
For Civil Engineering in this assessment, this comprises the following five specialised areas: 

♦ Materials and Construction: science of materials, steel, timber and masonry wall construction, construction oper¬ 
ation and management, construction informatics, building services engineering. 

♦ Structural Engineering: structural statics, earthquake engineering, maintenance management. 

♦ Geotechnical Engineering: foundation engineering, dam and tunnel remediation and construction, slope stabili¬ 
sation. 

♦ Hydraulic Engineering: water engineering and management, design of components and systems such as water 
supply systems and sewer networks. 

♦ Urban and Rural Planning: land planning, irrigation, traffic engineering, road and railway engineering, trans¬ 
portation networks, ecology and the environment, economics and sustainability, irrigation engineering, inland 
navigation engineering. 

Engineering Generic Skills 

i) The ability to use diverse methods to communicate effectively with the engineering community and with society 
at large. 

ii) The ability to demonstrate awareness of the wider multidisciplinary context of engineering. 

Engineering Processes 

1) Engineering Analysis 

i) The ability to apply knowledge and understanding to identify, formulate and solve engineering problems using 
established methods. 

ii) The ability to apply knowledge and understanding to analyse engineering products, processes and methods. 

iii) The ability to select and apply relevant analytic and modelling methods. 

iv) The ability to conduct searches of literature, and to use databases and other sources of information. 

v) The ability to design and conduct appropriate experiments, interpret the data and draw conclusions. 

vi) The ability to demonstrate workshop and laboratory skills. 

2) Engineering Design 

i) The ability to apply their knowledge and understanding to develop designs to meet defined and specified require¬ 
ments. 

ii) The ability to demonstrate an understanding of design methodologies, and an ability to use them. 

3) Engineering Practice 

i) The ability to select and use appropriate materials, equipment and tools. 

ii) The ability to combine theory and practice to solve engineering problems. 

iii) The ability to demonstrate understanding of applicable techniques and methods, and their limitations. 

iv) The ability to demonstrate understanding of the non-technical implications of engineering practice and commit¬ 
ment to professional ethics, responsibilities and norms of engineering practice. 

v) The ability to demonstrate understanding of the health, safety and legal issues and responsibilities of engineering 
practice, the impact of engineering solutions in a global, economic, societal and environmental context. 

vi) The ability to demonstrate knowledge of project management and business practices, such as risk and change 
management, and be aware of their limitations including project planning, labour, contracts, safety and health, cost 
analysis and control, professional ethics, subcontracting, environmental issues and information management, and 
management of construction and public works. 
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Table 2 An example of a constructed response task 


The Hoover Dam is a 221 -metre high concrete arch-gravity dam in the Black Canyon of the Colorado River in the United 

States of America. It was built to provide irrigation water, to control floods and to provide water for a hydroelectric power 

station at the base of the dam. 

1. Explain why this is a good dam site for hydroelectric power generation. You should discuss at least two aspects. 

2. Explain the two main design features that contribute to the structural strength and stability of the Hoover dam. 

3. The maximum electrical power generated by the turbines at the Hoover Dam is 2.08*10 9 . What is the approximate 
amount of water that flows through the turbines at this power output, if the power station operates at 90% efficiency? 

4. Imagine that a new dam is being planned today in a different location. Briefly explain two environmental effects of the 
dam (which could also be upstream or downstream) that an engineer would need to consider in an environmental impact 
statement. 


framework standards, shared by institutions that have been accredited by EUR-ACE authorised 
agencies, and covers the core concepts of the discipline in a comprehensive manner, but is suffi¬ 
ciently abstract so that it may apply to a diverse array of engineering programs. 

3.4. Articulating Concrete and Measurable Learning Outcomes 

Instrument development involves articulating concrete and measurable learning outcomes 
from abstract competences. The AHELO Feasibility Study provides an interesting example of 
how this can be done in assessment exercises, which can then be applied to designing courses 
and modules that constitute competence-based degree-programs. 

In the AHELO Feasibility Study instrument development, it was decided that “Basic and 
Engineering Sciences” would be measured mainly by multiple choice items, while “Engi¬ 
neering Generic Skills” and “Engineering Processes” would be measured mainly by constructive 
response tasks 3 . Multiple choice items aims to comprehensively and efficiently measure basic 
knowledge and skills, while constructive response tasks aim to measure how well students are 
able to “think” like an engineer in real world contexts 4 . 

Table 2 shows an example of the constructed response task. It focuses on a real world engi¬ 
neering structure, the Hoover Dam in the United States. The first question asks respondents to 
explain why the Black Canyon of the Colorado River is a good dam site for hydroelectric power 
generation. According to the scoring rubric, this question aims to measure the learning outcome 
“identifies the important aspects of selecting a site for power generation,” which represent the 
competences “basic and engineering sciences” and “engineering analysis.” Students are expected 
to provide responses discussing at least two from among the following six features: (a) dam 
height, high potential energy, (b) high flow rate, (c) lake capacity, (d) minimal social impact, 
(e) characteristic of the rock, and (f) narrow gorge. For each feature, examples of responses are 
listed to clarify the kinds of descriptions that can be given full or partial credit/points (OECD, 
2012 ). 

It is important to note that engineering experts were consulted multiple times to verify 
the instrument and scoring rubrics. For example, the feature “(d) minimal social impact” was 
originally not cited but was added to the scoring rubric upon request for from the Japanese 
team. Based on student responses from the 2011 small scale implementation prior to the 2012 
large scale implementation, Japanese faculty decided that students needed to be given credit for 
pointing out this aspect, since in a small and mountainous country like Japan, whether or not 
villages needed to be relocated to build the dam was in fact a major factor to be addressed in 
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selecting a dam site. Such dialogue among faculty proved to be a critical procedure for reaching 
consensus on the expected scope and level of learning outcomes. 

3.5. Generating Consensus through Scoring 

A total of 504 students in 12 universities participated in the 2012 large scale implementa¬ 
tion of the AHELO engineering assessment. Since the test was administered online, the multiple 
choice items were automatically scored by the system, while constructive response tasks were 
scored by trained scorers in each country. Training scorers required a systemic approach to 
ensure that all scorers were in full agreement with the interpretation of the scoring rubric. 

Lead Scorers of participating countries were invited to participate in two two-day inter¬ 
national scorer training sessions in November 2011 and March 2012. The training served two 
functions. The first function was to further modify the scoring rubric developed by the AHELO 
Consortium, making the scoring rubric more focused and, at the same time, more comprehen¬ 
sive. The second function was to enable Lead Scorers to reach agreement on the logic of scoring. 
Because scoring requires consensus on what kinds of responses can be identified as correct, the 
scoring exercise urged scorers to define precisely the scope and level of learning outcomes that 
students are expected to demonstrate. In effect, the training generated a clearer understanding of 
the scoring rubrics, as well as a sense of trust that scoring would be conducted in a consistent 
manner across countries. 

The scorer training within Japan took place immediately before the scoring session in June 
2012. Scorers consisted of 12 professors from 7 institutions. Two had prior experience in scoring 
during the 2011 small scale implementation, and others had acted as Test Administrators for the 
2012 large scale implementation. Civil engineering professors from the Tokyo Institute of Tech¬ 
nology also responded to our request for support. 

Under the leadership of the Lead Scorer, the 12 Scorers spent an afternoon and evening for 
training. After trial and error, it was decided that the best way to proceed was to work in two 
groups of six scorers. First, they would work individually on the training materials, and then 
work as a group to discuss why particular responses should be given particular scores. The Lead 
Scorer initiated the discussion and gradually shifted into a supervising role. The process was 
slow and controversial in the beginning, but eventually speeded up as scorers reached agreement 
on the general logic of scoring. This implies that once a shared understanding of the scope and 
level of bachelor-level learning outcomes are established, faculty is able to transfer the knowl¬ 
edge to new contexts and scenarios in a reliable manner, without having to collectively examine 
each learning outcome. 

The scoring took two full days, but proceeded smoothly. For the items that were double 
scored, the average reliability score for exact agreement reached 89%, indicating that for approx¬ 
imately nine out of ten student responses, the scorers gave exactly the same score. Through the 
scoring exercise, Japanese professors from diverse institutions were able to reach strong agree¬ 
ment on the scope and level of learning outcomes. 

3.6. Lessons Learnt from the AHELO Feasibility Study 

The AHELO Feasibility Study successfully brought faculty from institutions around the 
nation and around the globe in direct contact with each other, engaging in competence frame¬ 
work design, assessment instrument and scoring rubric development, and scoring. This action 
scheme proved invaluable in generating consensus among faculty regarding what bachelor-level 
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students are expected to know and be able to do, and in substantiating abstract competences into 
attainable and assessable learning outcomes. 

An important lesson learnt from the AHELO Feasibility Study is that faculty agreement on 
the scope and level of learning outcomes is transferable across scenarios. Once university profes¬ 
sors reach consensus on the general logic of scoring within a particular scenario, they are able 
to apply that logic to different scenarios and constantly reach high agreement in their scoring 
results. This means that occasional participation in external learning outcomes assessments may 
provide faculty long-lasting insights that will guide degree-program design and course plan¬ 
ning. Hence, learning outcomes assessment exercises can be a highly effective faculty develop¬ 
ment model for competence-based quality assurance. The AHELO Feasibility Study was a time 
consuming and costly experiment, but with rich implications for faculty development, the heart 
of systemic educational reform. 


Conclusion — Future Directions for Quality Assurance 

The purpose of this paper was to explore models of quality assurance in the age of higher 
education expansion. Review of the context of contemporary higher education showed that 
universities are under increasing pressure to be accountable for the competences of their grad¬ 
uates, but that this trend is in direct conflict with the autonomy and diversity of universities. 
Japanese higher education policy echoes the international focus on competences, but has not 
been able to provide the vision necessary to promote system-wide collaboration among faculty 
towards the development of competence-based degree-programs. Without a common framework 
to benchmark against, faculty currently lack the information necessary to determine whether 
their students are able to perform at the level expected by society. 

The AHELO Feasibility Study showed that the experience of participating in external 
learning outcomes assessments can help faculty in reaching substantive and tangible under¬ 
standing of the competence framework, and in substantiating attainable and assessable learning 
outcomes from the abstract competences. Such shared understanding is indeed the foundation of 
competence-based quality assurance. 

It is important to point out that assessment for faculty development and educational 
improvement is quite different from assessment for comparing and ranking students and institu¬ 
tions. Assessment designed to “compare” is guided by the principles of unbiasedness and reli¬ 
ability, and must try to measure the students’ abilities in a comprehensive and accurate manner 
for the comparison to be fair. For this purpose, instruments are often pre-tested and kept secure, 
limiting access from faculty and students. In sharp contrast, assessment designed to “inform” 
is guided by the principles of transparency and accessibility. Wide access to instruments and 
scoring rubrics should be guaranteed, and faculty workshops that link assessment outcomes with 
degree-program design and course planning should be encouraged. Needless to say, assessment 
as a model of competence-based quality assurance must be designed not to “compare” students 
or institutions against others in relative terms, but to “inform” faculty about student learning 
benchmarked against a common competence framework. 

Notes 

1. In Gakushi Rvoku, knowledge and understanding refers to the systemic understanding of basic 
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knowledge related to the individual’s field of study, as well as to the understanding of how that 
knowledge of the field of study is related to the existence of the self in relation to history, society, 
and nature. Generic skills for various purposes refer to the skills necessary for intellectual activi¬ 
ties, as well as professional and social life, including communication skills, mathematical literacy, 
information literacy, logical thinking, and problem solving skills. Attitude and propensity includes 
self-management, teamwork and leadership, ethics, social responsibility as citizens, and life-long 
learning skills. Finally, integrative learning experience and creative thinking refers to the ability to 
apply in a comprehensive manner the knowledge, skills, and attitudes acquired through learning in 
order to solve new problems. 

2. The AFIELO Consortium operated within the guidelines established by the fMFIE (OECD Fligher 
Education Programme) Governing Board and the AFIELO GNE (Group of National Experts), and 
under the management of the OECD secretariat. The Technical Advisory Group was established to 
provide international expertise and advice on operational and methodological issues. 

3. N1ER took the lead in proposing multiple choice items offered by the institution of Professional 
Engineers Japan, and the Japan Society of Civil Engineers. These items were judged to be in align¬ 
ment with the competence descriptions, and provided the advantage of having been tested by cohorts 
of Japanese engineering graduates. 

4. ACER took the lead in developing innovative constructed response tasks with advice from Austra¬ 
lian engineering experts. 
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